ostree.git
9 years agostatic-delta: Pretend that world unreadable objects are new objects
Mario Sanchez Prada [Thu, 15 Dec 2016 19:40:18 +0000 (19:40 +0000)]
static-delta: Pretend that world unreadable objects are new objects

This will prevent including in the delta the bits to update files that
are not world readable, so that we don't run into a permissions problem
when applying the deltas from a bare-user repository that has a bare
repository set as its parent.

This is the case for Endless when updating flatpak runtimes, as the
temporary directory created in ~/.local/share/flatpak/system-cache will
be of type bare-user with its parent set to /var/lib/flatpak which is a
bare repository in EOS, as it's shared with the one at /ostree/repo.

https://phabricator.endlessm.com/T14159

Closes: #634
Approved by: cgwalters

9 years agoRelease 2016.15
Colin Walters [Mon, 12 Dec 2016 16:54:03 +0000 (11:54 -0500)]
Release 2016.15

Closes: #632
Approved by: jlebon

9 years agoci: Rebase to f25
Colin Walters [Mon, 12 Dec 2016 14:05:06 +0000 (09:05 -0500)]
ci: Rebase to f25

This is now the devel target, plus I think this may fix some of the
ASAN issues I'm seeing; I mostly have been using f25 for local
testing.

Also remove the MAINTAINER line since the maintainers are defined by
`git log`.

Closes: #631
Approved by: jlebon

9 years agolib: Squash last use of GFile deltas_dir
Colin Walters [Fri, 9 Dec 2016 04:05:39 +0000 (23:05 -0500)]
lib: Squash last use of GFile deltas_dir

I was having this thought today about making more of the OS readonly,
and ultimately if we got to the point where all ostree operations are
through the repo and sysroot dfds, we could have rpm-ostree be the
only process holding those fds open, and have a read-only bind mount
on top.

Anyways, we're not there, likely won't be soon, but this gets us
closer to being fully fd relative.

Closes: #628
Approved by: jlebon

9 years agoci: Combine UBSAN and ASAN by default
Colin Walters [Thu, 8 Dec 2016 02:26:43 +0000 (21:26 -0500)]
ci: Combine UBSAN and ASAN by default

I only recently realized this was possible.  While we're still seeing
leaks in the CI environment for some reason, adding ASAN gives us
use-after-free detection etc., which is obviously still very useful
even if we're not doing leak checking.

Closes: #622
Approved by: jlebon

9 years agoci: Drop sudo installed tests
Colin Walters [Thu, 8 Dec 2016 18:35:29 +0000 (13:35 -0500)]
ci: Drop sudo installed tests

This conflicts with the ASAN work...and in general, I think I'd like
to make a new format for tests that require root, and have them be
defined to be in mutable containers or VMs.

Our coverage loss from this isn't much because some of these tests
already required `CAP_SYS_ADMIN` which we didn't have in Docker
anyways.

While we have the patient open, parallelize the regular installed
tests.

Closes: #622
Approved by: jlebon

9 years agotests: Tweak installed tests to deal with ASAN
Colin Walters [Thu, 8 Dec 2016 17:50:20 +0000 (12:50 -0500)]
tests: Tweak installed tests to deal with ASAN

We need to disable readdir-rand there too.

Closes: #622
Approved by: jlebon

9 years agoSkip gjs-based tests if ASAN is enabled
Colin Walters [Thu, 8 Dec 2016 02:59:40 +0000 (21:59 -0500)]
Skip gjs-based tests if ASAN is enabled

Unfortunately, introspection uses dlopen(), which doesn't quite
work when the DSO is compiled with ASAN but the outer executable
isn't.

Trying to inject LD_PRELOAD=libasan means the outer executable has to
be leak free...which, yeah, I'm not going to get into running ASAN
today on gjs or pygobject.

So, let's skip those tests - ideally, we still run them in some other
context without the sanitizers.  The coverage we have from them is
middling anyways.

Closes: #622
Approved by: jlebon

9 years agorepo: Add unconfigured-state to remote config options
Colin Walters [Thu, 8 Dec 2016 19:20:19 +0000 (14:20 -0500)]
repo: Add unconfigured-state to remote config options

This is a migration from the origin version.  It's
nicer to have it in the remote, since that's what one
needs to change.  Then tools don't need to mess with
the origin file.o

In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from.  We're just providing a better error.

Closes: https://github.com/ostreedev/ostree/issues/626
Closes: #627
Approved by: jlebon

9 years agobuild: Error if glib isn't found
Colin Walters [Thu, 8 Dec 2016 18:38:49 +0000 (13:38 -0500)]
build: Error if glib isn't found

This is a bit extracted from my work on ASAN.

Closes: #625
Approved by: jlebon

9 years agorepo: Fix annotations for remote_fetch_summary functions
Dan Nicholson [Fri, 9 Dec 2016 12:37:47 +0000 (06:37 -0600)]
repo: Fix annotations for remote_fetch_summary functions

These are out parameters, so add the (out) annotation and switch
(nullable) to (optional) since the latter is used for the purpose of
optional out parameters.

Closes: #629
Approved by: cgwalters

9 years agotree-wide: Switch to autoptr for GOptionContext
Colin Walters [Thu, 8 Dec 2016 15:32:09 +0000 (10:32 -0500)]
tree-wide: Switch to autoptr for GOptionContext

We were leaking in a few places that I noticed in an ASAN run.  Also,
this was one of the last non-autoptr cleanup sections we have in
`out:` cleanup sections, making us a lot closer to a potential
full-tree rewrite to `return FALSE`.

Closes: #624
Approved by: jlebon

9 years agobuild: Always do enum scanning now
Colin Walters [Thu, 8 Dec 2016 02:00:21 +0000 (21:00 -0500)]
build: Always do enum scanning now

Since we stopped including the libsoup headers in `ostree-fetcher.h`,
we can now unconditionally do enum scanning, and drop a build time
conditional.

Prep for libcurl porting.

Closes: #620
Approved by: jlebon

9 years ago[ASAN] sysroot: Squash a leak in lockfile acquisition
Colin Walters [Thu, 8 Dec 2016 15:01:35 +0000 (10:01 -0500)]
[ASAN] sysroot: Squash a leak in lockfile acquisition

I installed `parallel` in my dev container, which got me
the sysroot locking tests, which caught this leak when
built with ASAN.

Closes: #623
Approved by: jlebon

9 years agobuild: Make libsoup optional again
Colin Walters [Thu, 8 Dec 2016 02:16:52 +0000 (21:16 -0500)]
build: Make libsoup optional again

The "remote cookies" code broke this.  While I'm not sure anyone is
actually using ostree-without-http, it isn't too hard to keep the
build time conditional going.  Further, this work is preparatory for
libcurl porting.

Closes: #621
Approved by: jlebon

9 years agofetcher: Define an abstraction over SoupURI
Colin Walters [Tue, 6 Dec 2016 16:34:05 +0000 (11:34 -0500)]
fetcher: Define an abstraction over SoupURI

This is preparatory work for a potential libcurl backend.

Closes: #616
Approved by: jlebon

9 years agolib: Ensure we use _GNU_SOURCE in enum templates
Colin Walters [Tue, 6 Dec 2016 16:30:19 +0000 (11:30 -0500)]
lib: Ensure we use _GNU_SOURCE in enum templates

Due to the way glib-mkenums runs the preprocessor itself, it
doesn't pick up the `AC_USE_SYSTEM_EXTENSIONS()` that we have in
`configure.ac`.

This blew up in an obscure way when I later wanted to `#include
"libglnx.h"` in one of the headers, since it needs the `basename()`
from `string.h` which is only available with `_GNU_SOURCE`.

Closes: #616
Approved by: jlebon

9 years agobuild: Add more default errors
Colin Walters [Wed, 7 Dec 2016 14:55:10 +0000 (09:55 -0500)]
build: Add more default errors

Newer gcc has `-Wincompatible-pointer-types`, hooray!
Add a few others that we pass today.

Closes: #618
Approved by: jlebon

9 years agoci: Make all ci tests gating for Homu
Colin Walters [Tue, 6 Dec 2016 21:29:39 +0000 (16:29 -0500)]
ci: Make all ci tests gating for Homu

See the rhci docs.

Closes: #617
Approved by: jlebon

9 years agotree-wide: Use g_hash_table_add() where applicable
Colin Walters [Tue, 6 Dec 2016 02:06:43 +0000 (21:06 -0500)]
tree-wide: Use g_hash_table_add() where applicable

Just noticed a few while reading some code, decided to do a quick
cleanup.  It's shorter and clearer.

Closes: #614
Approved by: jlebon

9 years agolib: Remove unused ostree_metalink_get_uri()
Colin Walters [Tue, 6 Dec 2016 03:04:06 +0000 (22:04 -0500)]
lib: Remove unused ostree_metalink_get_uri()

While doing something else I noticed it was unused.

Closes: #615
Approved by: jlebon

9 years agopull: Write .commitpartial for local pulls first too
Colin Walters [Tue, 6 Dec 2016 01:56:55 +0000 (20:56 -0500)]
pull: Write .commitpartial for local pulls first too

This is what we do for non-local (i.e. HTTP) pulls; we wnat to
correctly handle being interrupted during partial pulls.

Closes: https://github.com/ostreedev/ostree/issues/579
Closes: #613
Approved by: jlebon

9 years agolib: Always checksum content in deltas
Colin Walters [Mon, 5 Dec 2016 22:22:46 +0000 (17:22 -0500)]
lib: Always checksum content in deltas

This is a follow up to conversation on list - in practice, if we're
backing away from summary signing, then it makes sense to remove the
special casing for checksums in deltas around summary signatures.

This is also related to the recent change to enable GPG checking for
commits in deltas - now we have a more coherent story between the
previous pull path and deltas.

I didn't do any performance checking, and while it's slightly annoying
that we're now doing sha256 on the delta content twice (once for the
part and once per object)...sha256 is pretty fast, I think most users
are I/O bound anyways, and it'd drop even farther if we started using
openssl.

Closes: #612
Approved by: jlebon

9 years agotests/keyfile-utils: Drop tests covering preconditions
Colin Walters [Mon, 5 Dec 2016 18:08:39 +0000 (13:08 -0500)]
tests/keyfile-utils: Drop tests covering preconditions

The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.

The coverage of this isn't worth the log spam basically.

Closes: #611
Approved by: jlebon

9 years agotests: Use G_DEBUG=fatal-warnings here too
Colin Walters [Mon, 5 Dec 2016 17:58:43 +0000 (12:58 -0500)]
tests: Use G_DEBUG=fatal-warnings here too

I am trying to track down a warning I see in `test-keyfile-utils`
which turned out to be the installed case only, but let's inject
this here too.

(The GLib default is broken, but it's hard to fix upstream without
 breaking the world)

Closes: #610
Approved by: jlebon

9 years ago[ASAN] tests: Cleanup all current remaining leaks
Colin Walters [Fri, 2 Dec 2016 18:45:04 +0000 (13:45 -0500)]
[ASAN] tests: Cleanup all current remaining leaks

We now run fully through ASAN here.

Closes: #609
Approved by: jlebon

9 years agoDefine and use cleanup helpers for libarchive
Colin Walters [Fri, 2 Dec 2016 18:34:32 +0000 (13:34 -0500)]
Define and use cleanup helpers for libarchive

This should fix some of the ASAN leaks around libarchive usage,
and is generally better.

Closes: #609
Approved by: jlebon

9 years agoTerminate individual tests after (10 * $TEST_TIMEOUT_FACTOR) minutes
Simon McVittie [Thu, 1 Dec 2016 16:28:27 +0000 (16:28 +0000)]
Terminate individual tests after (10 * $TEST_TIMEOUT_FACTOR) minutes

While using the Automake parallel test harness, if a test hangs for
long enough for an external watchdog to kill the entire build process
(as happens in Debian sbuild after 150 minutes with no activity on
stdout/stderr), the logs will not be shown. If we make an individual
test time out sooner, logs are more likely to be shown.

We use SIGABRT so that the process(es) under test will dump core,
allowing the point at which ostree is blocking to be analyzed.
After 1 minute, if any have not died, we kill them again with SIGKILL.

To support slow platforms and slow debugging tools, if
TEST_TIMEOUT_FACTOR is set, multiply the 10 minute timeout by that.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #607
Approved by: cgwalters

9 years agotests: prepend to an existing LD_LIBRARY_PATH, GI_TYPELIB_PATH
Simon McVittie [Wed, 30 Nov 2016 10:01:05 +0000 (10:01 +0000)]
tests: prepend to an existing LD_LIBRARY_PATH, GI_TYPELIB_PATH

If we're using LD_LIBRARY_PATH for some locally-built library, we want
to search those after OSTree's own libraries.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #606
Approved by: cgwalters

9 years ago[ASAN] tests: Fix leaks
Colin Walters [Tue, 29 Nov 2016 03:03:53 +0000 (22:03 -0500)]
[ASAN] tests: Fix leaks

Just for cleaner sanitizer output.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] set-origin: Squash a leak
Colin Walters [Tue, 29 Nov 2016 03:03:24 +0000 (22:03 -0500)]
[ASAN] set-origin: Squash a leak

Just a minor leak in the commandline.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] bootconfig: Drop a pointless strdup in parser
Colin Walters [Tue, 29 Nov 2016 03:02:42 +0000 (22:02 -0500)]
[ASAN] bootconfig: Drop a pointless strdup in parser

Not entirely sure how this was leaking, but anyways it showed
up in ASAN, and it's pointless to strdup here.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] metalink: Fix leaks of buffer
Colin Walters [Tue, 29 Nov 2016 03:01:33 +0000 (22:01 -0500)]
[ASAN] metalink: Fix leaks of buffer

We should be religious about the "only set output variables on
success", otherwise it makes leaks more likely.

But the real leak was us simply not using autoptr in one place.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] sysroot: Fix leak/double free of keyfile origin
Colin Walters [Tue, 29 Nov 2016 03:00:01 +0000 (22:00 -0500)]
[ASAN] sysroot: Fix leak/double free of keyfile origin

Use autoptr rather than manual cleanup.  The double free isn't a
security problem, since we trust origin files.

Closes: #598
Approved by: jlebon

9 years agotraverse: Use g_hash_table_add
Colin Walters [Tue, 29 Nov 2016 02:14:47 +0000 (21:14 -0500)]
traverse: Use g_hash_table_add

And "move semantics" via `g_steal_pointer()`.  Just a minor code
cleanup I noticed when I was hunting for a leak, which ended up being
elsewhere.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] cmdline: Fix minor leak in delta cmdline entrypoint
Colin Walters [Tue, 29 Nov 2016 02:12:53 +0000 (21:12 -0500)]
[ASAN] cmdline: Fix minor leak in delta cmdline entrypoint

Small, but it's important to stay clean.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] deltas: Fix minor memory leak
Colin Walters [Tue, 29 Nov 2016 02:12:23 +0000 (21:12 -0500)]
[ASAN] deltas: Fix minor memory leak

We were leaking the checksum, ensure we free it in both normal and
error paths.

Closes: #598
Approved by: jlebon

9 years ago[ASAN] delta compilation: More leak fixes
Colin Walters [Tue, 29 Nov 2016 02:11:37 +0000 (21:11 -0500)]
[ASAN] delta compilation: More leak fixes

Now that I remembered to do `env G_SLICE=always-malloc`, lots more
leaks become apparent.  Nothing major.

Closes: #598
Approved by: jlebon

9 years agoman: Mention bare-user in manpages, along with the other modes
Mario Sanchez Prada [Wed, 30 Nov 2016 15:41:38 +0000 (15:41 +0000)]
man: Mention bare-user in manpages, along with the other modes

Closes: #602
Closes: #603
Approved by: cgwalters

9 years agotravis-ci: document parameter variables
Simon McVittie [Tue, 29 Nov 2016 13:06:14 +0000 (13:06 +0000)]
travis-ci: document parameter variables

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: Enable stretch (the future Debian 9), replacing unstable
Simon McVittie [Mon, 28 Nov 2016 17:27:44 +0000 (17:27 +0000)]
travis-ci: Enable stretch (the future Debian 9), replacing unstable

My goal in building ostree for Debian unstable was that we would
have good coverage of "new code" paths. However, it was removed
for #571 as too much of a moving target. Debian testing is less of
a moving target, and in particular is always internally consistent
(packages are co-installable), which Debian unstable is not guaranteed
to be.

Debian 'stretch' is the future Debian 9, which should be released
next year.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: Use "slim" Debian image for testing
Simon McVittie [Mon, 28 Nov 2016 17:24:55 +0000 (17:24 +0000)]
travis-ci: Use "slim" Debian image for testing

Documentation and similar files are stripped from this image, making
it quicker to install.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: Run `make distcheck` too
Simon McVittie [Mon, 28 Nov 2016 17:24:08 +0000 (17:24 +0000)]
travis-ci: Run `make distcheck` too

ostree is now actively using that mode.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: cat the test log after successful test runs
Simon McVittie [Mon, 28 Nov 2016 17:23:43 +0000 (17:23 +0000)]
travis-ci: cat the test log after successful test runs

This lets us see which tests were skipped.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: Move helper function to before we start building anything
Simon McVittie [Mon, 28 Nov 2016 17:23:08 +0000 (17:23 +0000)]
travis-ci: Move helper function to before we start building anything

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: Use a non-ostree-specific name for the Docker image
Simon McVittie [Mon, 28 Nov 2016 17:22:24 +0000 (17:22 +0000)]
travis-ci: Use a non-ostree-specific name for the Docker image

This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agotravis-ci: put an explicit copyright/license on the scripts
Simon McVittie [Mon, 28 Nov 2016 17:21:46 +0000 (17:21 +0000)]
travis-ci: put an explicit copyright/license on the scripts

This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agoci-install: add ci_distro
Simon McVittie [Tue, 29 Nov 2016 13:05:57 +0000 (13:05 +0000)]
ci-install: add ci_distro

Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agoci-build: consistently use yes/no for booleans, not yes/empty
Simon McVittie [Mon, 28 Nov 2016 12:34:06 +0000 (12:34 +0000)]
ci-build: consistently use yes/no for booleans, not yes/empty

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agobuild: clean up ostree-remount if building without systemd
Simon McVittie [Mon, 28 Nov 2016 19:05:53 +0000 (19:05 +0000)]
build: clean up ostree-remount if building without systemd

This is necessary for "make distcheck" on Travis-CI.

Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters

9 years agodelta compilation: Fix leak
Alexander Larsson [Mon, 28 Nov 2016 16:21:59 +0000 (17:21 +0100)]
delta compilation: Fix leak

We need to ref-sik the new varian for g_autoptr to work

Closes: #597
Approved by: cgwalters

9 years agopull: Don't leak delta superblock variants
Alexander Larsson [Mon, 28 Nov 2016 14:57:11 +0000 (15:57 +0100)]
pull: Don't leak delta superblock variants

Closes: #596
Approved by: cgwalters

9 years agopull_with_options: Don't leak csum_v
Alexander Larsson [Mon, 28 Nov 2016 14:56:50 +0000 (15:56 +0100)]
pull_with_options: Don't leak csum_v

Closes: #596
Approved by: cgwalters

9 years agoostree-repo-traverse: Don't leak floating GVariant
Alexander Larsson [Mon, 28 Nov 2016 14:31:04 +0000 (15:31 +0100)]
ostree-repo-traverse: Don't leak floating GVariant

ostree_object_name_serialize returns a floating ref, so we need
to sink it before putting in the hashtable.

Closes: #595
Approved by: cgwalters

9 years agopull: scan_commit_object() - don't load variant twice
Alexander Larsson [Mon, 28 Nov 2016 12:58:19 +0000 (13:58 +0100)]
pull: scan_commit_object() - don't load variant twice

ostree_repo_load_commit already loaded the object, no need
to load it twice.

Closes: #595
Approved by: cgwalters

9 years agoostree-repo-traverse: Remove an accidental print statement
Jasper St. Pierre [Wed, 23 Nov 2016 21:52:39 +0000 (13:52 -0800)]
ostree-repo-traverse: Remove an accidental print statement

Closes: #594
Approved by: jlebon

9 years agoRelease 2016.14
Colin Walters [Wed, 23 Nov 2016 15:42:39 +0000 (10:42 -0500)]
Release 2016.14

Closes: #593
Approved by: jlebon

9 years agoFix broken link in docs/CONTRIBUTING.md
Abhay Kadam [Tue, 22 Nov 2016 13:04:52 +0000 (18:34 +0530)]
Fix broken link in docs/CONTRIBUTING.md

The link for git-rebase manual contains stray character (])
at the end.

Closes: #592
Approved by: jlebon

9 years agoostree-repo: Make the lock with a long-lasting FD
Jasper St. Pierre [Tue, 22 Nov 2016 00:05:55 +0000 (16:05 -0800)]
ostree-repo: Make the lock with a long-lasting FD

glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.

Closes: #591
Approved by: cgwalters

9 years agoostree-repo-static-delta-processing: Don't close(-1)
Jasper St. Pierre [Mon, 21 Nov 2016 23:10:24 +0000 (15:10 -0800)]
ostree-repo-static-delta-processing: Don't close(-1)

Ultimately harmless, but causes somewhat scary strace messages.

Closes: #591
Approved by: cgwalters

9 years agoostree-repo: Fix parameter name
Jasper St. Pierre [Mon, 21 Nov 2016 23:10:19 +0000 (15:10 -0800)]
ostree-repo: Fix parameter name

Closes: #591
Approved by: cgwalters

9 years ago[ASAN] lib: Squash various leaks in library and commandline
Colin Walters [Fri, 18 Nov 2016 15:33:50 +0000 (10:33 -0500)]
[ASAN] lib: Squash various leaks in library and commandline

The pull one is the most likely to affect users.  Otherwise mostly
just cleaning up `-fsanitize=address`.

Closes: #587
Approved by: jlebon

9 years ago[ASAN] tests: Fix some memleaks in libarchive importer
Colin Walters [Fri, 18 Nov 2016 15:32:59 +0000 (10:32 -0500)]
[ASAN] tests: Fix some memleaks in libarchive importer

Caught by `-fsanitize=address`.

Closes: #587
Approved by: jlebon

9 years agotests: Support TEST_SKIP_CLEANUP=err
Colin Walters [Fri, 18 Nov 2016 20:07:52 +0000 (15:07 -0500)]
tests: Support TEST_SKIP_CLEANUP=err

I find myself often wanting to debug interactively failing tests.
This makes it more convenient to keep around the temporary directories
just for those tests, rather than accumulating tons of tempdirs from
the successful tests as well.

Closes: #588
Approved by: jlebon

9 years agopull: Do GPG verify commit objects when using deltas
Colin Walters [Sun, 20 Nov 2016 21:17:22 +0000 (16:17 -0500)]
pull: Do GPG verify commit objects when using deltas

The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability.  Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.

Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.

Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have.  However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.

Closes: https://github.com/ostreedev/ostree/issues/517
Closes: #589
Approved by: jlebon

9 years agopull: Use new per-remote API for GPG verification
Colin Walters [Sat, 19 Nov 2016 01:23:13 +0000 (20:23 -0500)]
pull: Use new per-remote API for GPG verification

Trivial change, but makes things more obvious.  And we get test
coverage of the new API for free.

Closes: #589
Approved by: jlebon

9 years agopull: Dedup code for checking for > 0 valid results
Colin Walters [Sat, 19 Nov 2016 01:22:10 +0000 (20:22 -0500)]
pull: Dedup code for checking for > 0 valid results

We have a public API for this, let's use it internally.

Closes: #589
Approved by: jlebon

9 years ago[TSAN] Rework assertions to always access refcount atomically
Colin Walters [Thu, 17 Nov 2016 16:40:59 +0000 (11:40 -0500)]
[TSAN] Rework assertions to always access refcount atomically

`-fsanitize=address` complained that the `refcount > 0` assertions
were reading without atomics.  We can fix this by reworking them
to read the previous value.

Closes: #582
Approved by: jlebon

9 years ago[TSAN] main: Stop calling g_set_prgname()
Colin Walters [Thu, 17 Nov 2016 16:39:27 +0000 (11:39 -0500)]
[TSAN] main: Stop calling g_set_prgname()

It turns out this is basically racy with the presence of other
threads.  It was really cosmetic so let's stop doing it and make
`-fsanitize=thread` happy.

Closes: #582
Approved by: jlebon

9 years ago[UBSAN] deltas: Don't call memset(NULL, NULL, 0) with no xattrs
Colin Walters [Thu, 17 Nov 2016 18:48:58 +0000 (13:48 -0500)]
[UBSAN] deltas: Don't call memset(NULL, NULL, 0) with no xattrs

This is actually fine in practice, but it triggers this
`-fsanitize=undefined` warning I saw in the test suite log:

```
src/libostree/ostree-repo-static-delta-compilation.c:160:10: runtime error: null pointer passed as argument 1, which is declared to never be null
```

Closes: #584
Approved by: jlebon

9 years agoostree commit: Fix combining trees with multiple --tree=ref arguments
William Manley [Tue, 19 Jul 2016 02:14:26 +0000 (03:14 +0100)]
ostree commit: Fix combining trees with multiple --tree=ref arguments

You'd expect

    ostree commit --tree=ref=A --tree=ref=B

to produce a commit with the union of the trees given.  Instead you'd get
a commit with the contents of just the latter commit.  This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead.  This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.

This commit removes the optimisation, fixing the bug.  Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.

Closes: #581
Approved by: cgwalters

9 years agolib: Add an API to GPG verify a commit given a remote
Colin Walters [Wed, 16 Nov 2016 16:50:43 +0000 (11:50 -0500)]
lib: Add an API to GPG verify a commit given a remote

Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe

9 years agoAdd "gpgkeypath" option to remotes
Colin Walters [Wed, 16 Nov 2016 14:13:54 +0000 (09:13 -0500)]
Add "gpgkeypath" option to remotes

For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`.  It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.

This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.

Closes: https://github.com/ostreedev/ostree/issues/573
Closes: #575
Approved by: giuseppe

9 years agolib: Split out helper function to create GPG context
Colin Walters [Wed, 16 Nov 2016 14:10:39 +0000 (09:10 -0500)]
lib: Split out helper function to create GPG context

In prep for future work.

Closes: #575
Approved by: giuseppe

9 years agolib: Define and use cleanup functions for gpgme
Colin Walters [Wed, 16 Nov 2016 14:19:52 +0000 (09:19 -0500)]
lib: Define and use cleanup functions for gpgme

Just a cleanup in preparation for future work.

Closes: #575
Approved by: giuseppe

9 years agocommit: Fix reading xattrs from OstreeRepoFile:s
Alexander Larsson [Wed, 16 Nov 2016 21:46:45 +0000 (22:46 +0100)]
commit: Fix reading xattrs from OstreeRepoFile:s

When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.

Also adds a testcase for this.

Closes: #577
Approved by: cgwalters

9 years agopull: Redo logic for "scanning"
Colin Walters [Tue, 1 Nov 2016 17:51:55 +0000 (13:51 -0400)]
pull: Redo logic for "scanning"

What in the code is called "scanning" is ensuring (potentially
recursively) have an object, and if not, fetching it.  And then if
it's metadata, parsing it and finding new objects to fetch.

This logic has grown fairly complex.  What I'm trying to fix
right now is that if we're doing a pull-local to a remote repository
via `sshfs` (FUSE) we still end up scanning, which is inefficient.

We can take advantage of the "commitpartial" logic here - if a commit
isn't partial, it's complete, hence we don't need to scan it.

At the same time, I'm changing the logic here to *always* do scans for
dirtree objects.  This will fix cases where multiple commits share
dirtree objects.  We have "commitpartial" metadata, but no such concept
of partial/complete for dirtrees.

But, we'll only ever scan dirtrees if we scan commits, which is
what the section above fixes.

Closes: https://github.com/ostreedev/ostree/issues/543
Closes: #564
Approved by: alexlarsson

9 years agopull: Add support for `http-headers` option
Colin Walters [Wed, 16 Nov 2016 02:03:26 +0000 (21:03 -0500)]
pull: Add support for `http-headers` option

Some deployments may want to gate access to content based on things
like OAuth.  In this model, the client system would normally compute a
token and pass it to the server via an API.

We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.

Closes: #574
Approved by: giuseppe

9 years ago.redhat-ci.yml: no longer install libubsan & clang
Jonathan Lebon [Thu, 10 Nov 2016 17:56:37 +0000 (12:56 -0500)]
.redhat-ci.yml: no longer install libubsan & clang

Since they're now part of the auto-built image.

Closes: #572
Approved by: cgwalters

9 years agotravis: Drop debian unstable since we can't fetch packages reliably
Colin Walters [Thu, 10 Nov 2016 16:42:35 +0000 (11:42 -0500)]
travis: Drop debian unstable since we can't fetch packages reliably

I don't know what's going on, I suspect mirror churn.  Anyways,
it seems to be consistently failing now, so let's drop it.

Closes: #571
Approved by: jlebon

9 years agoRelease 2016.13
Colin Walters [Thu, 10 Nov 2016 16:27:43 +0000 (11:27 -0500)]
Release 2016.13

Closes: #570
Approved by: jlebon

9 years ago.redhat-ci.yml: use new build key
Jonathan Lebon [Wed, 9 Nov 2016 02:37:43 +0000 (21:37 -0500)]
.redhat-ci.yml: use new build key

This allows us to more concisely separate building from testing, which
in turn gives us a nicer inheritance pattern in our case.

See also: https://github.com/jlebon/redhat-ci/issues/11

Closes: #569
Approved by: cgwalters

9 years agodeltas: Only keep one file open at a time during compilation
Colin Walters [Thu, 3 Nov 2016 12:32:19 +0000 (08:32 -0400)]
deltas: Only keep one file open at a time during compilation

Otherwise it's possible for us to exhaust available file descriptors
or (on 32 bit) run up against mmap limits.

In the rollsum case, we didn't need to hold open the "from" object
at all.  And in the bsdiff case, we weren't even looking at either of
the files until we started processing.

Also, while we have the patient open, switch to using O_TMPFILE
if available.

Closes: #567
Approved by: giuseppe

9 years agoUpdate documentation for cookie handling commands
Sjoerd Simons [Mon, 17 Oct 2016 20:47:58 +0000 (22:47 +0200)]
Update documentation for cookie handling commands

Closes: #531
Approved by: cgwalters

9 years agotests: Add test for the cookie jar handling
Sjoerd Simons [Mon, 17 Oct 2016 20:53:32 +0000 (22:53 +0200)]
tests: Add test for the cookie jar handling

Closes: #531
Approved by: cgwalters

9 years agotrivial-httpd: Add support for checking cookies
Sjoerd Simons [Mon, 17 Oct 2016 20:35:40 +0000 (22:35 +0200)]
trivial-httpd: Add support for checking cookies

Allow passsing a list of cookie key/values to trivial-httpd which should
be provided to allow downloads

Closes: #531
Approved by: cgwalters

9 years agoOsreeFetcher: Treat 403 as not found
Sjoerd Simons [Mon, 17 Oct 2016 20:30:41 +0000 (22:30 +0200)]
OsreeFetcher: Treat 403 as not found

Private Cloudfront instances return 403 for objects which don't exist
rather then a 404.

Change the fetcher to assume 403 is ok for download that are "optional"
rather then erroring out at that step (e.g. trying to download a static
delta if the remote repo doesn't have those)

Closes: #531
Approved by: cgwalters

9 years agoremote: Add commands to add and remove cookies for a remote
Sjoerd Simons [Mon, 17 Oct 2016 20:30:14 +0000 (22:30 +0200)]
remote: Add commands to add and remove cookies for a remote

Add commands to add and remove cookies to a remotes cookie jar.

Closes: #531
Approved by: cgwalters

9 years agoremote: Add command to list cookies
Sjoerd Simons [Mon, 17 Oct 2016 20:14:14 +0000 (22:14 +0200)]
remote: Add command to list cookies

Closes: #531
Approved by: cgwalters

9 years agopull: Add per-remote cookie jar
Sjoerd Simons [Mon, 17 Oct 2016 19:39:38 +0000 (21:39 +0200)]
pull: Add per-remote cookie jar

Optionally read cookie jars for a remote to be used when downloading
data. This can be used for private repositories which require specific
cookies to be present, e.g. repositories hosted on Amazon cloudfront
using signed cookies.

Closes: #531
Approved by: cgwalters

9 years agopull: Don't do deltas with --commit-metadata-only
Colin Walters [Tue, 1 Nov 2016 20:39:58 +0000 (16:39 -0400)]
pull: Don't do deltas with --commit-metadata-only

We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.

I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.

Closes: https://github.com/ostreedev/ostree/issues/528
Closes: #566
Approved by: jlebon

9 years agorepo: Don't put remote refs in the summary file
Colin Walters [Tue, 1 Nov 2016 18:17:57 +0000 (14:17 -0400)]
repo: Don't put remote refs in the summary file

I was doing a chain of mirroring like A -> B -> C

And repo B had A as a remote.  When I added B as
a remote to C, the summary file of B had a ref
upstream:foo/bar/baz, which caused all pulls from
B to C to fail, since the summary file is only
expected to have refs, not refspecs.

Closes: https://github.com/ostreedev/ostree/issues/561
Closes: #565
Approved by: jlebon

9 years agoFilter bootloader supplied kernel cmdline options
Sjoerd Simons [Sun, 30 Oct 2016 20:06:27 +0000 (21:06 +0100)]
Filter bootloader supplied kernel cmdline options

Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.

Closes: #560
Approved by: cgwalters

9 years agoload_metadata_internal: don't leak GBytes
Simon McVittie [Sun, 30 Oct 2016 13:52:51 +0000 (13:52 +0000)]
load_metadata_internal: don't leak GBytes

Found by valgrind memcheck. g_variant_new_from_bytes takes a ref to the
bytes, so we need to release the original ref.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years agoFix some leaks of floating GVariants
Simon McVittie [Sat, 29 Oct 2016 17:47:11 +0000 (18:47 +0100)]
Fix some leaks of floating GVariants

ostree_repo_pull_with_options() and ostree_repo_remote_change() don't
sink floating GVariant arguments, and doing so now would be an
ABI change; so don't rely on them to do so.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years agoot_remote_builtin_show_url: autofree context
Simon McVittie [Thu, 27 Oct 2016 11:31:52 +0000 (12:31 +0100)]
ot_remote_builtin_show_url: autofree context

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years agoostree_repo_pull_with_options: clear dirs array
Simon McVittie [Thu, 27 Oct 2016 11:32:10 +0000 (12:32 +0100)]
ostree_repo_pull_with_options: clear dirs array

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years agokeyfile_set_from_vardict: free the string array
Simon McVittie [Sat, 29 Oct 2016 17:36:19 +0000 (18:36 +0100)]
keyfile_set_from_vardict: free the string array

g_variant_get_strv is (transfer container): the caller is expected to
free the array, but not the individual strings.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years ago_ostree_sysroot_write_deployments_internal: stop leaking hash table
Simon McVittie [Sat, 29 Oct 2016 17:37:09 +0000 (18:37 +0100)]
_ostree_sysroot_write_deployments_internal: stop leaking hash table

It appears the result of assign_bootserials() is never actually used,
but I haven't changed it to return void right now.

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters

9 years agoostree_sysroot_upgrader_finalize: free new_revision
Simon McVittie [Sat, 29 Oct 2016 17:37:32 +0000 (18:37 +0100)]
ostree_sysroot_upgrader_finalize: free new_revision

Leak found with valgrind memcheck.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters